Skip to content

Refresh competitor star counts and add benchmark caveat - #855

Merged
rohitg00 merged 4 commits into
mainfrom
docs/readme-refresh
Jun 7, 2026
Merged

rohitg00 merged 4 commits into
mainfrom
docs/readme-refresh

Conversation

@rohitg00

@rohitg00 rohitg00 commented Jun 7, 2026 •

Copy link
Copy Markdown
Owner

What

Refreshes the outdated competitor numbers in the comparison tables and adds a fairness note.

  • Comparison table (README + benchmark/COMPARISON.md): mem0 53K to 58K, Letta 22K to 23K, Khoj 34K to 35K, claude-mem 46K to 81K. Counts pulled live from GitHub.
  • Added a note under the README table: agentmemory's R@5 is measured on LongMemEval-S while the mem0 and Letta figures are their published LoCoMo numbers, a different dataset, so the columns are a ballpark not a head-to-head. This caveat already existed in benchmark/COMPARISON.md and is now surfaced where the table lives.

Verified, left unchanged

  • iii-engine pin: source pins v0.11.2 (IIPINNED_VERSION, iii-sdk dep), so the README pin note is accurate.
  • Per-agent hook counts (Claude Code 12, Codex 6, OpenCode 22) match source.
  • Test-count floor (1,390+) still holds.

Skill count (8 to 15) is handled by the detailed-skills PR, not duplicated here.

Summary by CodeRabbit

  • Documentation
    • Replaced competitor name "claude-mem" with "supermemory" across comparison guidance and feature matrix.
    • Added "MemPalace" to competitor coverage and guidance, with integration and benchmark caveats.
    • Updated approximate GitHub star counts and adjusted related table entries.
    • Clarified benchmark notes and added a disclaimer pointing to the benchmark comparison doc, distinguishing independently measured R@5 results from vendor self-reported figures.

Update mem0 (58K), Letta (23K), Khoj (35K), and claude-mem (81K) star counts in the README comparison table and benchmark/COMPARISON.md, and add a note that agentmemory's R@5 is LongMemEval-S while the mem0/Letta figures are LoCoMo (different dataset, ballpark only). Verified the iii-engine v0.11.2 pin and per-agent hook counts are current, so they are left unchanged.
@vercel

vercel Bot commented Jun 7, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
agentmemory Ready Ready Preview, Comment Jun 7, 2026 11:02pm

Request Review

@coderabbitai

coderabbitai Bot commented Jun 7, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: af16929c-af5c-4fd8-8a07-30b4a54f687d

📥 Commits

Reviewing files that changed from the base of the PR and between 41faa95 and d7570f8.

📒 Files selected for processing (1)
  • README.md
✅ Files skipped from review due to trivial changes (1)
  • README.md

📝 Walkthrough

Walkthrough

Updates README and benchmark/COMPARISON.md: replace claude-mem with supermemory, add MemPalace and other competitor columns, refresh approximate GitHub star counts and feature table cells, and add a README benchmark disclaimer clarifying dataset provenance and vendor self-reported figures.

Changes

Benchmark Comparison Updates

Layer / File(s) Summary
Replace competitor and update Feature Matrix
benchmark/COMPARISON.md
Replaced claude-mem with supermemory in the Feature Matrix, revised MemPalace LongMemEval row and the apples-vs-oranges caveat, and updated multiple matrix cells including GitHub star counts and feature attributes.
Token Efficiency & recommendation edits
benchmark/COMPARISON.md
Swapped claude-mem for supermemory in Token Efficiency and replaced the "Choose claude-mem if you want" block with "Choose supermemory if you want"; added a "Choose MemPalace if you want" block with integration caveats.
README competitor table and benchmark disclaimer
README.md
Expanded vs-competitors table with new columns (supermemory, MemPalace, Khoj, Hippo), updated approximate star-count headers and row values, revised the embedding-model/benchmark reference to benchmark/COMPARISON.md, and added a disclaimer about R@5 dataset provenance and vendor self-reported figures.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related PRs

  • rohitg00/agentmemory#807: Both PRs update README benchmark reporting—this PR adds an evaluation-source disclaimer for R@5 metrics while the related PR corrects P@5 metrics and benchmark notes.

Poem

🐰 The README hops, the tables gleam,
Columns swapped like a waking dream,
Stars nudged gently, notes made clear,
Memories sorted, benchmarks near,
Tiny edits — burrow cheers!

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately reflects the main changes: it updates competitor star counts across the documentation and introduces a benchmark caveat clarifying which results are independently measured versus vendor-claimed.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch docs/readme-refresh

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@benchmark/COMPARISON.md`:
- Around line 29-31: The docs still refer to "claude-mem" while the feature
matrix uses "supermemory": search for the string "claude-mem" and replace it
with "supermemory" in the competitor listing (specifically in the
section-competitors.svg content where the competitor enumeration text appears)
and in the German README translation (README.de-DE.md) so the terminology
matches the table header "supermemory"; verify the SVG text nodes/labels and any
README sentences are updated and preserved (escaping/encoding) and run a quick
grep to ensure no other files still contain "claude-mem".
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 8be09609-25dc-4749-8b06-abec458004dd

📥 Commits

Reviewing files that changed from the base of the PR and between a60ac01 and cc48aa6.

📒 Files selected for processing (2)
  • README.md
  • benchmark/COMPARISON.md
✅ Files skipped from review due to trivial changes (1)
  • README.md

Comment thread benchmark/COMPARISON.md Outdated
Comment on lines +29 to +31
| Feature | agentmemory | mem0 | Letta/MemGPT | Khoj | supermemory | Hippo |
|---|---|---|---|---|---|---|
| **GitHub stars** | Growing | 53K+ | 22K+ | 34K+ | 46K+ | Trending |
| **Type** | Memory engine + MCP server | Memory layer API | Full agent runtime | Personal AI | MCP server | Memory system |
| **Auto-capture via hooks** | ✅ 12 lifecycle hooks | ❌ Manual `add()` | ❌ Agent self-edits | ❌ Manual | ✅ Limited | ❌ Manual |
| **Search strategy** | BM25 + Vector + Graph | Vector + Graph | Vector (archival) | Semantic | FTS5 | Decay-weighted |
| **GitHub stars** | Growing | 58K+ | 23K+ | 35K+ | 26K+ | Trending |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Update cross-file references from claude-mem to supermemory.

The column replacement from claude-mem to supermemory in this Feature Matrix is not reflected in related documentation files, creating terminology inconsistencies:

  • assets/tags/section-competitors.svg:15 still lists "claude-mem" in the competitor enumeration
  • READMEs/README.de-DE.md:260 still references "claude-mem" in the German translation

These should be updated to maintain consistency across the documentation.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@benchmark/COMPARISON.md` around lines 29 - 31, The docs still refer to
"claude-mem" while the feature matrix uses "supermemory": search for the string
"claude-mem" and replace it with "supermemory" in the competitor listing
(specifically in the section-competitors.svg content where the competitor
enumeration text appears) and in the German README translation (README.de-DE.md)
so the terminology matches the table header "supermemory"; verify the SVG text
nodes/labels and any README sentences are updated and preserved
(escaping/encoding) and run a quick grep to ensure no other files still contain
"claude-mem".

…orted

Add MemPalace to the comparison (54K-star vector-only OSS) and label its ~96.6% R@5 as a vendor-published claim we have not reproduced. Clarify that only agentmemory's 95.2% is our own measured, reproducible result and every other number is a vendor claim on a different benchmark or harness, so the figures are ballpark not head-to-head.
Add Khoj, supermemory, MemPalace, and Hippo columns to the README comparison table (previously only mem0, Letta, and Built-in), matching benchmark/COMPARISON.md. Mark MemPalace and supermemory retrieval as vendor self-reported and unreproduced; only agentmemory's R@5 is our own measured result.
@rohitg00
rohitg00 merged commit 749c280 into main Jun 7, 2026
3 checks passed
@rohitg00
rohitg00 deleted the docs/readme-refresh branch June 7, 2026 23:04
healdigital pushed a commit to healdigital/agentmemory that referenced this pull request Jul 27, 2026
* docs: refresh competitor star counts and add benchmark methodology note

Update mem0 (58K), Letta (23K), Khoj (35K), and claude-mem (81K) star counts in the README comparison table and benchmark/COMPARISON.md, and add a note that agentmemory's R@5 is LongMemEval-S while the mem0/Letta figures are LoCoMo (different dataset, ballpark only). Verified the iii-engine v0.11.2 pin and per-agent hook counts are current, so they are left unchanged.

* docs: replace claude-mem with supermemory in comparison

* docs: add MemPalace column and mark competitor benchmarks as self-reported

Add MemPalace to the comparison (54K-star vector-only OSS) and label its ~96.6% R@5 as a vendor-published claim we have not reproduced. Clarify that only agentmemory's 95.2% is our own measured, reproducible result and every other number is a vendor claim on a different benchmark or harness, so the figures are ballpark not head-to-head.

* docs: expand README comparison table to full competitor set

Add Khoj, supermemory, MemPalace, and Hippo columns to the README comparison table (previously only mem0, Letta, and Built-in), matching benchmark/COMPARISON.md. Mark MemPalace and supermemory retrieval as vendor self-reported and unreproduced; only agentmemory's R@5 is our own measured result.

This branch was successfully deployed

1 active deployment
Preview — d7570f85 Deployed Jun 7, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant